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FOREWORD 



Educators have often been criticized for viewing their role merely a$ dispensers of knowledge 
and for showing a lack of concern with the application of this knowledge to their students' future 
lives. However, the general public has begun to accept the idea that the educational system has a 
responsibility to assist a// individuals in making orderly transitions to the world of work. Addition- 
ally, during the last decade, a number of innovative activities, projects, and programs have been 
developed at the federal, state, and local levels in which personnel are attempting to link education 
and work. Among these programs and projects are the following examples: career education, 
Experience-Based Career Education (EBCE), Part D exemplary projects and other projects in voca- 
tional education. Title IV-C of the Elementary and Secondary Education Act (ESEA), Fund for the 
Improvement of Post Secondary Education (FIPSE), and Title I and III of the Comprehensive 
Employment and Training Act (CETA). 

Personnel associated with education and work programs are becoming increasingly aware of 
the need for information related to accountability and needed improvements. Many reports— some 
informal, some in the literature-indicate a wide and exciting variety of approaches to evaluating 
programs that link education and work. However, the need exists for practitioners to become 
acquainted with evaluation ideas and materials available for particular situations. 

Recognizing these trends, the Education and Work Group of the National Institute of Educa- 
tion (NIE) contracted with the National Center for Research in Vocational Education (NCRVE) to 
develop the Career Education Measurement Series. The series includes five comprehensive "user 
oriented" handbooks intended to help local education personnel in measurement and evaluation. 
The handbooks in the series are: 

• Assessing Experiential Learning in Career Education 

• Career Education Measures: A Compendium of Evaluation Instruments 

• Improving the Accountability of Career Education Programs: Evaluation Guidelines and 
Checklists 

• A Guide for Improving Locally Developed Career Education Measures 

• Using Systematic Observation Techniques in Evaluating Career Education 

This handbook, A Guide for Improving Locally Developed Career Education Measures, is de- 
signed to provide practical guidelines for improving career education instruments developed at the 
local level. The handbook is primarily designed for people who lack training in evaluation and 
measurement but are responsible for this type of work. 

The Center is particularly indebted to Dr. N. L. McCaslin, Project Director and Dr. Jerry P. 
Walker, Associate Director for Evaluation at The Center, who prepared this handbook. Additionally, 
appreciation is extended to Dr. Michael Neuman, Assistant Professor of English at Capital University, 
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who edited the handbook, and to Dr. Kay Adams, Research Specialist, for reviewing developmental 
versions of the manuscript. A special note of thanks is extended to Dr. Ronald Bucknam who 
originally conceived this handbook series and continued his involvement through development as the 
Project Officer for the National Institute of Education. Valuable advice in the design and scope of 
the handbook series was received from an advisory committee composed of Dr. Robert Ebel, 
Michigan State University; Dr. Margaret Fergueron, State Director of Career Education in Florida; 
and Ms. Deede Sharpe, Georgia Department of Education. 

In an attempt to mak^ this handbook truly "user oriented," a user trial was conducted prior 
to publication of the handbook. The National Center is indebted to those career education practi- 
tioners who participated in the user trial. Their valuable assistance has greatly enhanced the utility 
of this handbook. These individuals included: 



Roland J. Cross 
Caroll Curtis 
Pat Doherty 
H. Stephen Hewlett 
Ellen Meister 
David G. Minnis 
Bernard Novick 
Harry E. Owen 
Carl W. Pepperm**n 
Allen I. Stessmarin 
Rose Trachtenberg 



Orono, Wisconsin 

Pennsylvania Department of Education 
New Jersey Department of Education 
Valparaiso, Indiana 
Madison, Wisconsin 
Edinboro, Pennsylvania 
Woodbridge, New Jersey 
Wirwihester, Kentucky 
Lewisburg, Pennsylvania 
Sheboygan, Wisconsin 
Hammond, Indiana 



Finally, a special note of appreciation is extended to Ms. Marlene Linton and Ms. Debi Roof 
who typed the manuscript of this publication. 



Robert E. Taylor 
Executive Director 
The National Center for Research 
in Vocational Education 
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SOME INTRODUCTORY REMARKS 



As a career educator, you probably have been asked to review evaluation instruments that 
others have developed and you may have wanted to improve instruments that you developed. And, 
at times, you may have even questioned your ability to actually critique and improve these instru- 
ments. A number of questions come to mind when developing evaluation instruments that "meas- 
ure up." How can you be sure that you have developed or adapted an effective evaluation instrument? 
How can you be confident of its effectiveness? How can you demonstrate the accuracy of its results? 
This handbook has been designed to assist you in answering these and other questions and thus im- 
prove the quality of your career education evaluation instruments. 

You may ask, "Why is evaluation necessary?" Now, more than ever, evaluation is necessary 
to demonstrate an educator's accountability to the public. In 1976, the National Center for Educa- 
tional Statistics estimated that educational institutions were expending more than $120 billion annually. 
Of this total, expenditures for elementary and secondary schools accounted for more than $75 billion. 
Naturally, people wonder what the schools are doing with these tax dollars. This is especially true if 
they assume (as reported by a 1976 Gallup Poll) that declining scores on standardized tests mean that 
the quality of education today is diminishing. Therefore, even with the widespread support for 
career education, educators such as yourself need to provide evidence of what students are achieving. 
In addition there is another factor that makes evaluation so important: evaluation should also pro- 
vide information for improving the planning and development of career education programs. This 
type of information should enable you to do a better job of meeting the needs of students and to 
make sound educational decisions. 

In order to demonstrate levels of student achievement, administrators and teachers have relied 
upon testing or measurement devices. In the case of career education, the lack of appropriate meas- 
ures has, in many instances, caused individuals to develop their own instruments or to modify those 
developed by someone else. However, in most cases, administrators and teachers are not trained or 
experienced in test development and consequently lack confidence in the instruments that they 
have adopted. 

The handbook has been designed to enable you to develop reliable instruments for evaluating 
your career education programs (at least before you consult with experts). This handbook provides 
basic information that should enable you to adapt and improve existing career education measure- 
ment devices. These improvement efforts have been organized around the following six topics: 
{^) Planning and Designing, (2) Checking Reliability, (3) Determining Validity, (4) Eliminating 
Stert^otypes, (5) Devising the Format, and (6) Gauging Readability. A brief description of these 
topics is presented below. 

• "Planning and Designing" explains how to structure revision efforts. The unit constitutes a 
"blueprint" for revising career education instruments. 

• "Checking Reliability" reviews the important components of dependability and consistency 
in a career education instrument. 
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• "Determining Validity" poses the question, "Are you really measuring what you think 
you are measuring?" If you cannot tell what the measure is assessing, obviously the 
measure is of little value. 

• "Eliminating Stereotypes" presents important ways to prevent the instrument from inhibit- 
ing responses because of an individual's sex or ethnic background. 

• "Devising the Format" explains how the layout of the instrument can enhance the accuracy 
of the results from the instrument. 

• "Gauging Readability" analyzes the ability of a career education instrument to communicate 
clearly with those for whom the device is intended. Often the "message" sent is not the 
"mesage" received. 

The sequence of these six topics is not important. Each section presents central problems and 
some practical suggestions for minimizing them and improving the career education instrument. 
Simply select those of greatest relevance to your career education instrument. The following check* 
list has been developed for rating how your career education instrument "measures up" and selecting 
those topics that are most relevant for improvement efforts. 



How Does Your Career Education Instrument "Measure Up"? 

It is important to keep in mind that the higher the "quality" of a career education instrunwtt, 
the more likely it is to capture information of which you can be confident. The lower the "quality" 
of a career education instrument, the less you can be confident of the information collected. Figure 
1 presents a checklist you can use to assess your career education instruments. The checklist pro- 
vides questions, response categories, ar>d handbook references. The questions included on the check* 
list are keyed to the six major topics of the handbook. The handbook references cite the handbook 
pages containing the source material for the questions. The response column will enable you to rate 
your instrument for each of the questions asked. 

In using the checklist, first read the question and mark your response in the appropriate space 
at the left of the question. If you are unsure of your response, refer to the column on the right arKl 
review the pages that are cited. Then, total your responses for each of the categories (i.e., Yes, No, 
Not Sure, or Not Applicable) and record each total in the space provided at the end of the check- 
list. Once you have totaled your responses, turn to page 6 to see what your responses mean. 




A CHECKLIST FOR RATING HOW YOUR CAREER 
EDUCATION INSTRUMENT "MEASURES UP" 



Ym 


Np 


Not 


Not 




Quwttoni 


Rata 
(Fail 


ranoi 

») 












PLANNING AND DESIGNING 
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1. 


Did you rtvitw tht go«ls and obitctivts for th« c«r«tr 
fducttion topic to bt asstsMd? 


9 


-10 














2. 


Did you dtvtlop an outlint of th« conttnt %f%m to bt 
includad in vour inttrumtnt? 


11 


-12 










3. 


Did you dt ttmnina the clMtts of studant bthivior for 
th« appropriatt aducational domain (t.g.* cognitivt. 
afftctivt. psychomotor) that would ba assasatd by 

tha instrument? 


13 


-ie 














4. 


Did you chtck tha relationship of aach itam to tha 
ctassas of tha appropriata aducational domain and 
contant araas to ba asscssad? 


13,17. 












CHECKING REUASILITY 








— 







5. 


Did you datarmina tha tast*rttast coaffJ^rtant? 


20 


• 22 














6. 


Did you detarmirta tha intarnal consistency coafficiant? 


22 


-24 










7. 


Did you datarmina tha aquivalanca coafficiant? 


24 


26 










8. 


Did you chack for ttams that naarly avaryooa missad? 


27 












9. 


Did you check for items that nearly everyone Qot 

correct? 


27 
















10. 


Did you check the correct arnj ir>correct responses for 
the higher scoring irnjividuals? 


28 


29 










11. 


Did you check the correct and incorrect respc^nses 
for the lower scoring ifKjividuals? 


28 


-29 










12- 


Did you consider the length of time avatlabte for 
testing? 


29 












13. 


Did yOL check the directions for clarity? 


29 












14. 


Did you check the directions for conciseness? 


29 












15. 


Did you avoid the use of ambiguous terms? 
DETERMINING VALIDITY 


29 


■32 










16, 


Did you determine the content validity of the 


34 





instrument? 
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Figurt 1 (continiMd) 



.. H^fidbooli 
Not Not Qutttiora Rfftf«nc« 

No Suf« Applidblt (Pa9M) 



DETERMINING VALIDITY (conttnu«d) 

17 Did you determine the concurrent validity of the 34 
instrument? 

18 Dtd you determine the predictive validity of the 34, 36 
instrument? 

19. Did you include examples of a wide variety of careers 36 • 37 

and occupations in the items? 

20 Did you collect information about occupations to 36 - 37 
help improve the realism of the items? 

21 Did you originally develop twice as many items as 37 
you intefKjed to use in the instrument? 

22 Did you ha^e kitowixigeable people review the 37 
instrunrwnt for rtievance to the topic being assessed? 



ELIMINATING STEREOTYPES 

_ 23. Did you include plural or both masculine and feminine 41 

pronouns in the items to avoid sexual stereotypes? 

24 Did you use neutral terms in describing various occu* 41 

pations or careers? 

25 Did you use names in the instrument that were 42 

different fronf> Anglo in origin? 

26 Did you include items that reflect a variety of life 42 

styles (e.g , single parent, families, extended families, 
suburban, urban, rural, etc )? 

27 Did you include items that reflect a variety of person* 42 

alities (e.g., outgoing, shy, verbal, nonverbal, optimistic, 
pessimistic, confident, insecure, etc )? 

28 Did you include items that reflect a variety of socio* 42 

economic levels? 

29 Did you include items that reflect a variety of cultures? 42 

30. Did you include some items that reflect individuals with 42 

handicapped conditions? 

31 Did you include some items that reflect males in 42 

traditionally female roles and/or occupations? 

32. Did you include some items that reflect females in 42 

traditionally male roles and/or occupations? 
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Figure 1 (continued) 



^BESSa Handbook 

Not Not Qirattiom Rtf«r«nG« 

Yn No Surt Appltoiblt (Rtgtt) 

DEVISING THE FORMAT 

33. Did you limit the different items on your instrunnent 43 

to two or three types (e.g.. multiple-choice, true- 
false. Ltkert. etcj? 

34. Did you use large type in prepenng the instrument? 43 

■ 35. Did you use clear illustrations in preparing the 43 

instrument? 

36. Did you keep all possible responses to an item on the 43-44 

same page as the question? 

_ 37. Did you prepare an attractive page layout? 44 

GAUGING READABILITY 

38. Did you begin with relatively easy items? 45 

39. Did you test the instrument for roadability? 45 

40. Did you pilot test the instrument with individuals 45 



similar to those for whom the measure was interxied? 



TOTALS 
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A high total of "yes" responses suggests that you have a "good" instrument. Just for fun, let's 
determine a "grade" for your instrument; the following grading scale is based upon the total number 
of "yes" responses that you obtain: 

37 or more A (Move to the top of the class) 

33 to 36 = B (Better than most) 

30 to 32 = C (Not bad-but there is room for improvement) 

26 to 29 = D (You had better reconsider) 

25 or Itrts = F (Have you thought about starting over?) 
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PLANNING AND DESIGNING 



Planning provides you with an opportunity to carefully examine the purpose, content and use 
of a career education instrument. Yet, as a general rule, planning receives insufficient attention 
when an instrument is being revised. Prior to revising an instrument, you need to have a clear picture 
of what you wish to assess. Take a few minutes to reflect on your instrument; describe the purpose 
of the instrument in as direct a manner as possible; think about the items (questions) that you have 
included; consider the relationship of the individual items to the instrument's purpose. Do all of 
your items relate to the purpose of the instrument? You may find it necessary to eliminate or re* 
vise some items that you had originally included or to add additional items. This section of the 
handbook will discuss some of the common problems encountered in planning career education 
instruments and some practical ways for revising these instruments. 



What Are Some of the Problems Encountered When Attempting 
to Improve the Planning and Designing of Career Education 
Instruments? 

There are a number of problems that impede the planning and designing of career education 
instruments. First, the real purpose of the instrument is often not clearly established. You need to 
question the reason for the instrument and the items it contains: 

• Is the instrument designed to assess students' specific knowledge? 

• Is the instrument designed to assess students' ability to make judgments? 

• Is the instrument designed to determine students' ability to perform certain tasks or to 
achieve certain skills? 

• Is the instrument designed to assess some combination of the skills listed above or even 
to serve other purposes? 

Career education instruments often test trivial matters when insufficient time has been devoted 
to developing the items on the test. Items that ask simply for recall of certain facts or figures are 
generally easier to develop than are those that test for understanding or ability to apply specific 
knowledge. 

Items are frequently borrowt j .om previously developed tests. Utilizing existing items is 
easier than developing new items, especially when you are under pressure to develop a test for im- 
mediate use. However, when items are taken from existing tests, it is often difficult to relate them 
sufficiently to the goals, outcomes, or elements of your career education program. 

We are often unaware of the various uses that can be made of tests. Katz (1973) identified 
eight different uses of test scores. You may have other uses that you would like your instrument 
to serve. Examples of Katz's uses of test scores as they apply to career education are presented 
on the following page. 
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Placing students — More and more emphasis is being placed upon open-entry and open- 
exit types of educational activities. In career education, there is a need to consider 
where a student is, in relationship to a particular activity rather than assume that 
all students must start at the same point and spend the same amount of time on a 
given activity. For example, a student with previous work experience may not 
need the same amount of information on job-seeking skills as one who has not had 
work experience. Therefore, one possible use of a career education instrument on 
job-seeking skills could be to place a student in a group requiring similar information 

Diagnosing pupils' \waknesses — It is also possible to use tests to identify specific 

strengths and weaknesses of individuals. Let us assume that you want to determine 
the particular aspects of work attitudes with which students need assistance. After 
administering a test of work attitudes, you might find that some students have a 
high understanding of the need to be loyal to their employers and have positive 
attitudes toward putting in a day's work for a day's pay. While other students have 
a low understanding of the importance of being consistent and persistent. In this 
case, you could use an individual's score on various subparts of a work attitude test 
in prescribing activities to increase identified deficiencies. 

Assessing effectiveness of instmction — A question often asked of an educational 
program is, does it work? People are often reluctant to adopt or adapt new 
practices without evidence as to their effectiveness. In this case, a test could 
be used to compare individuals' occupational knowledge prior to career education 
activities with their self-concept after the activities. The resulting difference could 
then be used as an indication of the effectiveness of career education. 

Predicting pupils' achievement — A test might also be used to predict pupils' achieve- 
ment at some later point in life. For example, some might argue that students 
who are knowledgeable of the variety of potential careers available to them 
make better career choices, and, subsequently, are more satisfied with their 
employment. In this case, students' scores on a test of career opp>ortunities 
could be compared with a test of their job satisfaction at some later point in 
time. The scores could then be compared to determine the accuracy with which 
students' scores on a career awareness test did, in fact, predict their satisfaction 
with their employment. 

Evaluating schools — In many districts, student achievement scores are routinely 
compared by schools and buildings. This allows administrators to see how 
various schools are doing both within the district and in comparison to other 
districts such as theirs. Often this type of information is used in releases to 
the school board as well as to the general public. A comparison of average scores 
by school, on a career development test, would be an example of this type of 
test usage. 

Comparing a pupil's achievement to a fixed standard — The public is becoming more 
and more concerned that students are unable to perform what they consider 
''basic life skills." Although many arguments (both pro and con) can be pre- 
sented, it appears that some schools are establishing minimal levels of students' 
knowledge, attitudes, and skills. For example, a person could be required to 
complete a job application blank prior to being advanced or graduated. In this 
case, a student could be given an application form and asked to complete it, or 
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respond to specific questions related to application forms. In either case, a minimum 
score could be established and students would be required to perform at this level 
prior to advancement or graduation. 

Comparing a pupil's achiavemant to a score on anothar tast — A student's time is one of 
the limited resources with v^ich educators must work. Consequently, this time 
must be used as efficiently and effectively as possible. In career education, we 
should not spend time obtaining evaluation information that might already be 
available. Therefore, test scores of the various aspects of career education should 
be compared with other test scores to see if there are any tests that might be 
eliminated. To illustrate, if scores on a career planning test are highly correlated 
with scores on a career information test already in use, it may be possible to drop 
one of the tests from the testing and evaluation program. 

Comparing a pupil's achiavemant to a teacher's marks — A final use that can be made 
of test scores is to compare the results of an achievement test with the marks of 
a teacher. In many cases, teacher's marks are thought to reflect many interper- 
sonal characteristics of the student and are not truly indicative of knowledge, 
attitude, or skills of the student. Therefore, an achievement test is often thought 
to be a more objective estimate of a pupil's achievement than are a teacher's marks. 
In career education, for example, an assessment of work 'attitudes via an indepen- 
dently developed test would generally be viewed more credibly than would teacher 
rankings of students' work attitudes. 

We have noted a few of the complications you might encounter in planning and revising instru- 
ments. So to forestall these problems, we will now discuss some practical ways of improving the 
way that career education instruments are planned and revised. 



How Can I Improve the Planning and Designing 
of Career Education Instruments? 

Several techniques are available to those who wish to improve the planning and designing of 
career education instruments. This section develops the following four suggestions: 

1. Before revising the instrument, review information on the goals and objectives of the 
career education topic to be assessed. 

2. Develop a basic outline of what is to be included in the instrument. 

3. Determine the types of student behavior associated with the area to be tested. 

4. Develop a planning matrix. 

1. Reviewing information on the goals and objectives. In order to improve the quality of an 
instrument, you should review the goals and objectives of the career education topic to be assessed. 
Despite the fact that research in career education is complicated by the absence of definitions, you 
should review the goals, elements, and objectives that shaped your career education program. In 
addition to your own goals and objectives, you may want to review the goals, elements, and outcomes 
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of other career education programs.* Other resources for researching the topic you wish to assess 
range from the Educational Research information Center (ERIC) to the card catalog in your local 
library. And don't hesitate to consult the numerous authorities available, such as your state career 
education director or personnel from planning, research, and evaluation units. Other knowledgeable 
individuals are located in colleges and universities, educational laboratories and centers, and private 
agencies. Larger school districts may also have planning, research, and evaluation units that would 
be helpful to you in this process. 

But remember your program has its own goals and objectives and these must be kept foremost 
as you revise your career education instruments. As you review the goals and objectives of the 
career e *'ication topic you wish to assess, it will be helpful to prepare a chart that illustrates the 
relationships of the objectives to the goals. A sample of this type of chart is presented in Figure 2. 



Figure 2 

AN EXAMPLE OF THE RELATIONSHIP 
BETWEEN GOALS AND OBJECTIVES 



S«mplt Go«ls of 
a Career Educetion 
Program 




Sample Obiectives of a 
Career Education Program 


1 To increase the 


1 


Students will be able to locate potential employers 


ability of students 






to obtain employ- 


2 


Students wH) be able to neatly and accurately complete an application 


ment 




for employment 




3 


Students will demonstrate desirable behaviors during an interview 






Situation 


2 To understand how 


1. 


Students will be able to identify situations with career related problems 


one makes career 






related decisions 


2 


Students will be able to list the steps followed in solving career related 






problems. 




3 


Students will distinguish between the causes and symptoms of career 






related problems 



^Developmental Program Goals for the Comprehensive Career Education Model (1972); 
Michigan Career Education: Career Development Goals and Performance Indicators (1974), Basic 
Learner Outcomes for Career Education (1973); Perspectives on the Problem of Evaluation in Career 
Education (1976), Evaluation and Educational Decision Making. A Functional Guide to Evaluating 
Career Education (1975). 
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2. Developing a content outline. As mentioned previously, it is essential that you have a clear 
picture of what you are trying to assess before you begin work on the instrument. An outline will 
help you to clarify your purpose. 

Let us suppose that you are developing a test on "applying for a job." One of the first deter- 
minations you should make is whether this topic is logically connected to the goals, outcomes, or 
elements of your entire project. For example, if one of your goals is to help students obtain em- 
ployment, the logical relationship of "applying for a job" to the goal is quite clear (see Figure 2). 
Often, however, there are discrepancies between topics and goals that should be detected before 
imprecise items are devised. Furthermore, relating your topic to your goal will help you to generate 
additional information or a broadened outline. For example, applying for a job is one of four skills 
related to obtaining employment that the state of Florida (1973) identified in the following portion 
of an outline: 

• Obtaining Employment 

A. Choosing an occupation 

B. Locating potential employers 

C. Securing initial information about opportunities 

D. Applying for a |ob 

You may agree or disagree with this conceptualization of obtaining employment; however, the 
important point is that there should be a test of the logical connection between the topic and the 
goals or outcomes for the career education program. 

Another step in generating the outline is to consider subheadings; for example, what activities 
are involved in "applying for a job"? Again, the state of Florida (1973) determined that the topic 
of applying for a job consists of the following content areas: 

General procedures for applying for a job 

Job interviews: general interviewing procedures 

Job interviews: manners and appearance 

Job interviews, communication skills 

Application forms and resumes 

Preemployment testing 

These content areas can then be used as major sections, around which the individual items can be 
developed. An example of how these content areas can be used m developing a planning framework 
for a test on applying for a job is presented in Figure 3. If each of these content areas is of equal 
importance, t*ie number of items for each section should be the same. If some of the content areas 
are more important than others then they should contain more items than those of lesser importance. 
These content areas also form one dimension of the planning matrix presented in Figure 7. 
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Figurt 3 

A PLANNING FRAMEWORK FOR TEST ON APPLYING FOR A JOB TEST 



Applying for a Job 



to 



G«n«ftl 

Proc«dum 
for Applying 
for a Job 



Job Inttr- 
vMwt: Gan«ral 
Intarvitwing 
Proc«durtt 



22 



Tttt 

Ittmt 



Job litter- 
vi«wt: Mann«n 
and 



Tatt 

I tarn t 



Job Intar- 
vitwt: Com- 
rminicstion 
Skills 



Tatt 
Itamt 



Application 
Format 

and 
Ratumas 



Tatt 
Itamt 



Pra- 
amploymant 
Tatting 



Tatt 
Itamt 



Tatt 
Itamt 
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3. Relating student behavior. Bloom (1956) classif ied student behavior into three domains: 
cognitive, affective, and psychomotor Recalling these domains v\^ill help you to state precisely and 
realistically the desired outcomes for students. After these domains are surveyed, they will be com- 
bined with the content areas listed above to form a matrix of factors to consider when planning 
career education instruments. 

JhB cognitive domain includes skills that require the recall or recognition of knowledge and 
the development of intellectual abilities. The cognitive domain contains the following six major 
classes: knowledge, comprehension, application, analysis, synthesis, and evaluation. These classes 
are arranged in order from the simple to the complex. Definitions of each of these classes and 
examples of career education assessment activities are presented in Figure 4. For further informa- 
tion related to this domain, you may want to review the Taxonomy of Educational Objectives: The 
Classification of Educational Goals, Handbook I: Cognitive Domain (Bloom, et al., 1956). 

The affective domain includes behavior that reflects an individual's feelings, emotions, or 
degree of acceptance or rejection. The affective domain includes the following five classes: receiv- 
ing, responding, valuing, organizing, and characterizing by a value or value complex. Again, these 
classes are arranged from the simple to the complex. Definitions of each of these clams and exam- 
ples of career education assessment activities are presented in Figure 5. For further and more de- 
tailed information on the affective domain, you may want to study the Taxonomy of Educational 
Objective: The Classification of Educational Goals. Handbook II: Affective Domain (Krathwohl, 
et al., 1956). 

1 he psychomotor domain includes actions that require some muscular or motor skill, some 
manipulation of material and objects, or some act which requires neuromuscular coordination. The 
psychomotor domain includes the following four classes: gross bodily movements, finely coordinated 
movements, nonverbal communication behaviors, and speech behaviors. For additional information 
on this domain, you are encouraged to review Behavioral Objectives and Instruction (Kibler, et al., 
1970). 

Although these three domains have been presented as mutually exclusive categories, deter- 
mining the proper classification of a given behavior is not always a simple task. Furthermore, the 
classes within each domain often overlap, making it even more difficult to make the proper classifi- 
cation. Therefore, when trying to classify items according to educational domains and classes you 
will probably have to make some arbitrary decisions regarding their "best fit." 

4. Utilizing a planning matrix. You can develop a matrix that will help you examine the com- 
position and number of items for each topic to be included in an evaluation instrument. In such a 
matrix, the classes of the appropriate domain are placed along the left hand side of the matrix and 
the content areas are placed across the top of the matrix. It should be noted that it would be quite 
unusual to include measurement related to all three (cognitive, affective, and psychomotor) of the 
domains in the same instrument. As a result of using a planning matrix, you can revise your career 
education instruments to more adequately assess the aspects of career education that you deem to 
be most important. Figure 7 presents a matrix that was developed to examine an instrument on 
"applying for a job." 

The six content areas included in "applying for a job" that were identified earlier are placed 
across the top of the matrix. Similarly, the six classes of the cognitive domain that we discussed 
earlier are placed down the side of the matrix. As a result, the matrix contains a total of 36 differ- 
ent combinations of the classes of the cognitive domain of educational behavior and the content 
areas included in "applying for a job." Each of these combinations represents one cell in our 
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COGNITIVE DOMAIN CLASSES AND EXAMPLE 
CAREER EDUCATION ASSESSMENT ACTIVITIES 
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Examples of Career Education 
Assesvnent Activities 



Knowledge— Remembering, either 
by recognition or recall, of ideas, 
nnaterial, or phenomena. 

Comprehension-Understanding what 
is being communicated and being able 
to make some use of the material or 
ideas contairted in it. 



Application— Requires the act of 
problem solving, using acquired 
knowledges arxJ understandings. 
The problems should be new wnd 
different from the situations in 
which material was originally 
presented. 

Analysis— The ability to break 
down material into component 
parts. 



Synthesis— Putting elements and 
parts together so as to form a 
whole. 

Evaluation— Making judgenr^ents 
about the value, for some purpose 
of ideas, works solutions, methods, 
material, etc. 



Ask students to identify tools or equipment 
that worken in a particular occupation 
would be expected to use. 

After readir^g about an occupation, have the 
students resporKi to a set of questions about 
the occupation or prepare a short essay on 
their ability to work in the occupation and 
interest in the occupation. 

Provide an example of a personal conflict 
in a work setting and ask the students how 
they would minimize the conflict 



Provide a scenario of someone selecting a 
new job and ask the students to distinguish 
relevant arguments for the job from irrelevant 
arguments for the job. 

Ask students to prepare a written response 
to a letter in which an individual complains 
about unsatisfactory service. 

Ask the students to rank potential applicants 
for a particular job based upon: (1 ) a com- 
pleted application and (2) vita. 
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Figures 



AFFECTIVE DOMAIN CLASSES AND EXAMPLE 
CAREER EDUCATION ASSESSMENT ACTIVITIES 



Classes 



Examples of Career Education 
Assessment Activities 



Receiving— Deals with 
a learner's willingness 
to learn. 



Ask students to indicate likes and dislikes related 
to various careers or willingness to study about 
careers. 



Responding— Is concerned 
with learner's level of 
commitment to active 
participation. 



Ask students about their feeling of satisfaction as 
it relates to various types of work that they have 
done. 



Valuing— Emphasizes the 
development of attitudes, 
beliefs, and values as it relates 
to the worth of a thing, pheno- 
menon, or behavior. 



Ask students to indicate whether they agree or 
disagree with statements regarding various work 
habits. 



Organizing— Requires the 
learner to bring together 
values to form a value 
system. 



Describe a life style and four possible attitudes 
related to the life style. Ask students to arrange 
the attitudes in order of preference. 



Characterizing by a Value or 
Value Complex-Refers to an 
organization of attitudes and 
values that have been inter- 
nalized into a consistent 
systpnn or philosophy which 
guides the individual. 



Provide students with a list of different occupations. 
Next, pair each occupation with each of the remain- 
ing occupations on the list. Then ask students to 
indicate which one in the paired occupations on the 
list is most indicative of the type of work they would 
like to do. 
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PSYCHOMOTOR DOMIAIN CLASSES AND EXAMPLE 
CAREER EDUCATION ASSESSMENT ACTIVITIES 



CImm 



Examplti of CtrMr Education 
Activitias 



Grow Bodily Movamant- 
It charactarizad by 
movamant of antira 

limbt. 



Explain to studantt that at \mrkan in haaith 
caraars thay may ba raquirad to mova patiants 
from ona araa to aoothar. To aMts itrangih, 
aik ttudants to mova an objact or waight 
from ona point to anothar. 



Finaly Coordinatad 
Movamanti— I nvolvai 
movamaf*c of tha 
axtramitiat, a.g., hands, 
fingan, faat, ate. 



Giva itudants handwrittan Itttar and aik 
tham to typa tha lattari within a givan tima 
frama. 



Nonverbal Communication 
Bahaviors— Cantars on facial 
expressions, gestures, and/or 
bodily movements. 



Set up an interview setting, and ask students 
to demonstrate erchusiasm without actually 
speaking. 



Speech Beheviors— Relates 
to an individual's ability 
to verbally express himself/ 
herself. 



Describe a work situation and ask students 
to "state" the consequences of poor work 
habits. 
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A POSSIBLE PLANNING MATRIX FOR A CAREER EDUCATION 
INSTRUMENT ON APPLYING FOR A JOB 



CtWMOf 

Domain 


CofitMit Afao 


Qanaral 

Pfooa^tefai 
for Apply- 
kHifora 
Job 


Job llllM^ 
arti Intar- 

SSL- 


Job Intaf* 
vla^^ws 
Ntemaff 
and 

^^paManaa 


Joblntar* 
vlaavt: 

OoiMIIU* 

niaation 
SkiHt 


aatlofi 

FOfRII 

and 

RaiiHnat 


Pra- 

amf^lo'yfnant 
T«tii« 


Totil 
of 

hami 


^MToant 
of 

Tem 

ItMM 




1 


1 


2 


1 


1 


1 


7 


14 




1 


1 


2 


1 




4 


« 


18 


Applicstton 




1 




1 


S 




7 


14 


AMiysis 










ft 




1^ 




Syntfmis 


5 


1 




1 




1 


8 


ft 


Evtiuction 


1 


1 


1 


1 


1 




5 


10 


Total Numbar 
of Itarm 


8 


8 


8 


8 


12 


6 


50 


100 


Parcant of 
Total Itams 


16 


16 


16 


16 


24 


12 


100 
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planning matrix. In examining a test, each item is read and classified according to the class of edu- 
cational domain and content area that it represents. Once this determination has been made, a dash 
mark is made in the appropriate cell of the planning matrix. Once the examination of the test is 
completed, the number and percent of items for each class and content area can be computed. 

It should also be noted that not all of the cells in the matrix will necessarily have entries. 
Figure 7 indicates that a total of 50 items were included in the test; that number appears in the 
total-number-of items cell for the matrix. Look again at Figure 7, you will note that the most 
important content area is application forms and resumes; and preemployment testing is the least 
important content area. Similarly, the analysis class is the most important and evaluation is the 
least important class of the cognitive domain. 

By using a planning matrix in revising an instrument, you will be able to determine the relative 
emphasis for each of the classes of domains and content areas included in your test. Without using 
a planning technique such as a matrix, you could unintentionally place emphasis on certain classes 
or content areas and thus develop an instrument that does not correctly measure what is intended. 
This is often the case when instruments are developed utilizing only items that are easy to write 
and score. 



18 




CHECKING RELIABILITY 



In describing a penon at reliabia, you might ny the ptnon is trustworthy, dependtbls, ttMbin, 
and consisttnt; and you would bt confidant of tha parson's ability to perform. Tha tarm "raliabil* 
ity" as applied to an evaluation instrument has much the same meaning. In other words, a reliable 
instrument will provide similar results each time it is used. It is important that you be confident of 
the results obtained with evaluation instruments you use. Otherwise, it will be impossible for you 
to rely on the results. Therefore, you need to determine the reliability of your test prior to plaeir^ 
too much confidence in your results. 

R<Nlability estimates can be established according to the different formulas found in such basic 
meesurement texts as Cronbach's Es99ntm/$ ofPfychologial T§iHng (1970) and Ebel's Mmuirir^ 
Education^ Achmv9m9nt (1965). As a general rule, reliability is expressed as a decimal number 
(coefficient) ranging from 0.(X) to ^1 .(X). The closer the decimal (coefficient) is to ^1 .00, the more 
reliable the test is said to be. Figure 8 suggests general guidelines to follow in interpreting the ade- 
quacy of reliability coefficients. 

Figures 

SUGGESTED GUIDELINES FOR INTERPRETING 
RELIABILITY COEFFICIENTS 



Reliability Coefficients Adequacy 



.90 and higher " Super 

.80 - .89 « Not bad 

.70 - .79 « OK, but . . . 

.60 - .69 * Better watch out 

Less than .60 = No way! 



Katz (1973) indicated that there are three basic kinds of reliability: stability, internal consist- 
ency, and equivalence. Each of these types will be discussed in this section. 

Stability (or test retest reliability) is the likelihood that an instrument will elicit the same levels 
of performance at different testing times. Test-retest reliability is determined by comparing pretest 
scores with postscores. 
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Intmrml consistency is the similarity of scores on different parts of a test that are taken at the 
same time. As a general rule, internal consistency is determined by correlating the score on odd- 
numbered items with the score on even-numbered items. 

Equivalence is the correlation between scores on different testing forms of the same instrument. 
Equivalence is determined simply by comparing scores on two or more separate forms of the same 
instrument. 

The purpose of this section is not to develop the capability to compute complex reliability 
indices for instruments; rather, it is to show how easily locally developed instruments can be im- 
proved through paying attention to reliability issues. 



How Can I Ottermint the RetiabilitY of 
Career Education Instruments? 

There are a number of reference works that analyze in-depth the statistical and mathematical 

theory involved in determining reliability. This section, however, is designed to provide more sim- 
plified methods for determining the three kinds of reliability. 

StebHity (test-retest) is probably the easiest kind of reliability to determine because it only 
requires administering an instrument two times to the same group of students and then comparing 
the scores. Nevertheless, several problems need to be recognized. The most obvious one is that the 
students tend to remember the questions and answers from one administration of the instrument to 
the next. Another problem is that students often feel that taking a test twice is a waste of their 
time; this attitude alone can lessen the reliability of the results. Still another problem is that if the 
time between the two administrations is too long, the students' responses may differ because of 
learning that occurred between the two administrations. 

One way to determine the test-retest reliability could be to administer your instrument imme- 
diately prior to and after a school vacation lasting for one or two weeks. This would eliminate the 
effect of any instruction that might occur between the two administrations of the instrument. 

Recognizing the fact that these problems exist, let's consider the following situation. Let's 
assume that you have developed an instrument to assess students' work attitudes. Let's also assume 
that, in an attempt to eliminate (or minimize) the effect of learning that could occur between ad- 
ministrations of the test, you administered the test immediately before and immediately after spring 
vacation. 

You will recall from page 19 that the closer the reliability coefficient is to +1.00, the more 
reliable the instrument is considered to be. You can compute the "stability coefficient" of your 
instrument using the Kendall rank correlation coefficient T (tau). Siegel (1956) suggests the follow- 
ing formula for tau: 



In Siegel's formula, S = the number of individuals on the second administration of the vy^rk 
attitude test that are in the same order as they were on the first administration of the work attitude 
test, and N = the number of students which were ranked on both administrations of the work atti- 
tude test. To determine the value of tau, start by arranging the students' names m sequential order 



T = 



_S 

%N (N-1) 
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from the highest score to the lowest score for eech of the two separate administrations of the test 
(as shown in Figure 9). 

Figure 9 

RANK ORDER OF STUDENTS ON THE WORK ATTITUDE TEST 



RMuh of First Adminittrstion Rssult of Smmd Administration 



1. 


KAwrfsne 


1. 


MaHana Laa 


2. 


Trenton Bsyes 


2. 


Jose Qonzalas 


3. 


Kay Angont 


3. 


Kay Angona 


4. 


Josa Gonialet 


4. 


Trenton Bayes 


S. 


Crais Garrttt 


5. 


Katll Hall 


e. 


Jarry Wone 


e. 


Paulina JaooblnskI 


7. 


Kalli Hall 


7. 


Craig Garratt 


8. 


Pauline Jaeoblnskl 


8. 


Jarry Wong 


9. 


Barry Wlloox 


9. 


Barry WIIodx 


10. 


Rlehard Alston 


10. 


Jade Vanoa 


11. 


Martin Stona 


11. 


jaanana Lang 


12. 


Jack Vanoa 


12. 


Martin Stona 


13. 


Jaanttta Lang 


13. 


Mm7 Ching 


14. 


Mary Chlng 


14. 


Stava Nelson 


15. 


Stavt Nation 


15. 


Richard Alston 



Next, compile a table like the one shown in Figure 10. Examine Figure 10, note that there 
are three columns. The first column lists the names of students in order (from highest to lowest) 
according to their scores on the first administration of the test. Column 2 lists their respective 
rank on the first administration of the test and column 3 lists their rank on the second administra- 
tion of the test. 

Figure 10 

AGREEMENT BETWEEN RANKINGS 
ON THE WORK ATTITUDE TEST 



Column 1 


Column 2 


Column 3 


Individual 


Rank on First 


Rank on Sacond 




Administration 


Administration 


Mariana Lm 


1 


1 


Trenton Bayat 


2 


4 


Kay Angona 


3 


3 


Jota Qonzalas 


4 


2 


Craig Garratt 


5 


7 


Jarry Wong 


6 


8 


Kalli Hall 


7 


5 


Paulina Jacobintki 


8 


6 


Barry Wilcox 


g 


g 


Rkrhard Alston 


10 


IS 


Martin Stona 


11 


12 


Jack Vanct 


12 


10 


Jaanttta Long 


13 


11 


Mary Ching 


14 


13 


Stav« Nalton 


15 


14 
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You are now ready to compute S, which in turn will enable you to determine the degree of 
agreement between the scores received on the two administrations of the test. 

To find the value of S, start with the rank of the first student on the second administration 
of the test and count the number of ranks below it which are larger; then subtract from this num- 
ber the number of ranks below it that are smaller. After completing this process for all ranks, sum 
the results to obtain S. Thus, in our example the rank which is listed first on the second adminis* 
tration is 1 . This rank has 14 ranks which are larger below it and 0 ranks which are smaller. There- 
fore, its contribution to S is (14-0). The next rank is 4. It has 1 1 ranks below it that are larger and 
2 ranks that are smaller so that the contribution to S is (1 1-2). By proceeding in this manner you 
can see that: 

S = (14-0) + (11-2) + (1M) + (11-0) +(8-2) + (7-2) + (8-0) + (7-0) + (6-0) + (0-5) 
+ (2-2) + (3-0) + (2-0) + (1-0) = 77 

Knowing that 15 students were administered the work attitude test, we can proceed to com- 
pute the stability coefficient. 



y^ (15) (15-1) 

T= 77 
105 

T= .73 

In this case the stability coefficient is only .73. According to Figure 8, this test is "OK, 
but. . . Before placing complete confidence in this test it should be made more reliable. Pages 
26-32 present ways to improve reliability. 

Internal consistency (or split-half reliability) can be determined by performing the following 
three steps. First, divide the test into two parts (e.g., on the basis of odd- and even-numbered 
items). Then, score each part separately, as if it were the entire test. Finally, compute an internal 
consistency coefficient to determine the degree of agreement between the two scores. 

Consider this example: let us assume that you have developed a fifty-item test gauging knowl- 
edge of the world of work. Divide the test according to odd- and even-numbered items into two 
halves of twenty-five items each. Score each half as you would any other test. Then (as in Figure 
1 1 ) record the two scores for each individual and determine the rank order of these individuals. 

In computing the internal consistency coefficient you can use the same basic procedure that 
was suggested for determining the stability coefficient. First, arrange the names in rank order of 
their scores on the even-numbered items. Second, place the rank order of each student on the 
even-numbered items m a column to the right of the name. These numbers should be in sequential 
order from one to the number of students that took the test (in this case 15). Third, place the rank 
order of each student on the odd-numbered items in a second column to the right of the first col- 
umn. An example of this procedure is presented in Figure 1 2. 

Again, the procedure suggested by Siegel (1956) can be used to compute Kendall's rank cor- 
relation coefficient (tau) for the internal consistency coefficient: 
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SCORES AND RANKS FOR INDIVIDUALS ON 
A TEST OF KNOWLEDGE OF THE WORLD OF WORK 



Sooraon tUMtant Rmk loonon StudMitRank 



IndhrUhMl 


Evan Half 


ontha 
Evan Half 


OM)Wf 


on ttM 
Odd Half 


Richard Alston 


72 


14 


75 


14 


Kiy Angoru 


89 


B 


90 


5 


Trtnton Bayts 


63 


IB 


OB 


15 


Mary Ching 


90 


4 


88 


6 


Craig Garrttt 


86 


8 


86 


8 


Jose Gonzales 


91 


3 


02 


4 


Ktlll Hall 


83 


11 


83 


11 


Paulina Jtcoblnski 


92 


2 


03 


3 


Jeanatta Lang 


82 


12 


04 


2 


Marlar^a Laa 


93 


1 


06 


1 


Stava Nalson 


85 


9 


82 


12 


Martin Stona 


80 


13 


81 


13 


Jack Vanca 


87 


7 


86 


9 


Barry Wilcox 


88 


6 


84 


10 


Jarry Wor>g 


84 


10 


87 


7 



Figurt 12 

AGREEMENT BETWEEN RANKINGS ON 
THE KNOWLEDGE OF THE WORLD OF WORK TEST 



Column Cdumn 2 C d y mw 3 

Nama Rank on Evan- Rank on Odd- 

ffvuniiiafaa nam iviiniDafaQ iTams 



Marlane Lm 


1 


1 


Pauline Jacobinski 


2 


3 


Jos« Gonzalet 


3 


4 


Mary Ching 


4 


6 


Kay Angona 


5 


5 


Barry Wilcox 


6 


10 


Jack Vance 


7 


9 


Craig Garrett 


8 


8 


Steve Nelson 


9 


12 


Jerry Wong 


10 


7 


Kelli Hall 


11 


11 


Jeartette Lang 


12 


2 


Martin Stone 


13 


13 


Richard Altton 


14 


14 


Trenton Bayer 


15 


15 
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In this analysis, S the number of individuals on the scoring of the odd-numbered items that 
are in the same order as their score on the even-numbered items arxj N « the total number of stu- 
dents who took the knowledge of the world of work test. In determining the value of S, start with 
the rank of the first student on scoring of the odd-numbered items arnl count the number of ranks 
below it that are larger. Then, subtract from this number the number of ranks below it that are 
smaller. Do this for each of the entries in Column 3 and sum the totals to obtain S. To illustrate, 
the rank of the first student of the odd-numbered scoring is 1 . Below this rank are fourteen ranks 
that are larger and none that are smaller. Therefore, the component of the S value for this rank 
would be 14-0 or 14. Similarly for the second student, there are twelve ranks larger than 3 and one 
that is smaller than 3. Hence, the component for this entry would be 12-1 or 1 1. Continue this 
process and you can see that: 

S» (14-0) + (12-1) + (11-1) + (9-2) + (9-1) + (5-4) + (5-3) + (5-2) + (3-3) + (4-1) + (3-1) 
+ (3-0) + (2-0) + (1-0) = 67 

Again, since fifteen students were administered the knowledge of the world of work test, 
you can compute the internal consistency coefficient by substituting this value for N: 



^/^(15) (15-1) 

T= 67 
105 



T= .64 

Refer again to Figure 8; you will note that this coefficient is marginal and the Internal consisterKy 
of the test should be questioned. Therefore, you should not place much confidence in the results 
of this test. Pages 26-32 present methods for improving internal consisterKy. 

Equivalence is determined by comparing an instrument's results with the results of a different 
test of the same material. Let us assume that tvw> different teachers have each prepared a test de- 
signed to measure interpersonal relationships of ninth grade students, and you are interested in 
determining whether both instruments provide similar results. To measure the equivalence, you 
would first administer both instruments to the same group of students. Then you would rank order 
the students' names according to their scores on each test, with the highest scoring student listed 
first and the lowest scoring student listed last (as in Figure 13). 

Once the rank order for each student on each of the two tests has been determined, you are 
ready to determine the agreement between the rankings on the two tests. To help in making this 
determination, compile a table like that shown in Figure 14. Arrange the names in rank order of 
their scores on Test A and list the names in Column 1 . Place the number of the rank order in 
Column 2. These ranks should be in sequential order from one to the number of students who 
took the test (in this case ten). Next, place the rank order of each student on Test B in Column 
3. 

You are now ready to compute the equivalence coefficient following the procedure suggested 
by Siegel (1956) for the Kendall rank correlation coefficient or tau (T) where: 
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RANK ORDER OF STUDENT PERFORMANCES ON 
TWO TEST$ ON INTERPERSONAL RELATIONSHIPS 



Interpvnonal Rtlationships Intwptnonal Rtittionships 



Test A 


TaitB 


1. Pauline Jacobinski 


1. Martin Stone 


2. Martin Stona 


2. Mary Ching 


3. Mary Ching 


3. Paulina Jaoobinaici 


4. StavaNalaon 


4. Steve N«taon 


5. Josa Gonzalai 


5. KelliHall 


6. KalliHall 


6. TrentCNi Bayea 


7. Tranton Bayas 


7. JoaaGonnlaa 


8. Mariana Laa 


8. Jerry Wong 


9. Jarry Wong 


9. Mariana Lee 


10. Craig Garrett 


10. Craig G^Tftt 



Figur* 14 

AGREEMENT BETWEEN RANKINGS ON TWO TESTS 
ON INTERPERSONAL RELATIONSHIPS 



Column 1 


Column 2 


Column 3 


Individual 


Rank Order 


Rank Order 




on Teft A 


onTaMB 


1. Pauline Jacobinski 


1 


3 


2. Martin Stone 


2 


1 


3. Mary Ching 


3 


2 


4. Steve Nelson 


4 


4 


5. Jose Gonzales 


5 


7 


6. Kelli Hall 


6 


5 


7. Trenton Bayes 


7 


6 


8. Marlene Lee 


8 


9 


9. Jerry Wong 


9 


8 


10. Craig Garrett 


10 


10 
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(N*1) 



As was the c«S9 in our two similar analyses, S " the number of individuals on the scoring of 
Test B that are in the same order as their scx>res on Test A and N « the total number of studems 
who took both Test A and Test B. In computing the value of we take the rank of the first stu- 
dent on Test 8 and determine the number of ranks bek>w it that are larger and subtract from that 
number the number of ranks below it that are smaller. Similarly, we proceed through the rankings 
of remaining students on Test 8. Hence, you can see the following computation: 

S - (7-2) + (8-0) + (7-0) + (6-0) + (3-2) + (4-0) + (3-0) + (1-1) + (1-0) - 35 

In this case, the total number of students taking both Test A and B was ten. Therefore, we can sub- 
stitute into our formula accordingly and see that: 

T. 35 

% (10) (9) 

T, 35 
45 

T- .78 

In this case, we should be cautious (see Figure 8) in stating that these tests are equivalent. Each 
test should be carefully examined to determine which test more accurately reflects the desired 
measurement. The following section presents ways to improve the reliability of evaluation instru* 
ments that you develop. 

In this section we have discussed the overall reliability of your career education instruments. 
As a result of studying this section, you should be able to determine if the reliability of your career 
education instrument needs to be improved. The section that follows indicates what you can do 
with specific sections and items to improve the reliability of your career education test. 



How Can I Improve the Reliability of 
Career Education Instruments? 

The improvement of an instrument's realibility is one evaluation activity from which career 
education personnel can receive immediate and direct benefits without having to rely upon other 
educational agencies for materials. This section presents six practical suggestions for improving the 
reliability of your career education instruments: 

1 . Remove the items from the instrument that everyone misses and those that everyone 
gets correct. 

2. Remove the responses that no one selects and those that everyone selects. 

3. Retain items on which the higher scoring individuals were successful and the lower 
scoring individuals were unsuccessful. 

4. Increase the length of the test. 
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5. Provide deer and concise directions to the individual taking the tett. 

6. Eliminate ambiguous items. 

The first three suggestions liAed above are particularly applicable to career education instru- 
ments that are designed to discriminate highly knowledgeeble individuals from lees knowledgeeble 
individuals. Testsof this type are called norm-referenced. Norm-referenoed testing compares indi* 
vidua! performance of a student against the perfonmance of a particular group of students. Responses 
to a norm-referenced test are generally distributed in a nonnal ''bell shtped'' curve. An example of 
the "bell shaped" curve is shown in Figure 15. 




From a test development point of view, you would not want to include items which the stu- 
dents with low knowledge levels get correct and the students with the high knowledge levels get 
incorrect. The remainder of this section will briefly discuss each of the six suggestk>r)s mentioned 
previously. 

1 . Removing items that everyone answers correctly or ans we n in c orrec tl y. A reliable instru- 
ment will identify persons who have achieved expected career education outcomes, so if an item on 
the instrument is answered the same way by everyone, the item does not help to distinguish pvr- 
formances and thus fails to contribute to the instrument's reliability. The process of isolating these 
items is simple. Review the instrument item by item, recording for each item, the number of per- 
sons who answered it incorrectly. The person scoring the instrument coukJ record the number 
directly on the instrument. Or the teacher could ask the students to indicate (by a show of hands) 
who got the item right or wrong; furthemrK>re, discussing the reasons for the various responses will 
elicit information that could then be used to develop more reliable items. 

2. Removing the responses that no one or everyone s e l e cts . As described above, any item that 
does not help to distinguish persons who have achieved an outcome from those who have not should 
be eliminated from the instrument. For example, let us consider the following item: 
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Which of the following jobs rBquires the most education? 

a. collage professor 

b. accountant 

c. garbage collector 

d. teacher 

Suppose that an analysis of 100 responses to the item revealed the following pattern: 

a. 20 individuals 

b. 40 individuals 

c. 0 individuals 

d. 40 individuals 

Obviously, response "c" is not effective in helping to distinguish among student reactions; therefore 
"c" should be replaced by a more realistic response. Discussing the item with students might germ- 
rate a more rwlistic alternative (for example, ''stock broker"). The instrument should then be 
readministered. and the number of responses should once again be tabulated for each item. Let us 
imagine that the individual responses for this item would then be: 

a. 30 individuals 

b. 20 individuals 

c. 15 individuals 

d. 25 individuals 

All of the responses now appear to be more realistic, and they all help distinguish individuals who 
responded correctly from those who didn't. Therefore, the substitution would make the instrument 
more reliable. 

3. Retaining items that differentiate higher and lower scoring individuals. Another way to 
increase the reliability of an instrument is to include only those items that the higher scoring indi* 
viduals answer correctly and the lower scoring individuals answer incorrectly. To determine these 
items, arrange the students by scores with the highest score first and the lowest score last. Next, 
divide these students into four equal groupings so that you can place the highest 25 percent into 
one group and the lowest 25 percent into another group. Then examine the instrument item by 
item as shown in Figure 16. 

Figure 16 

A COMPARISON OF HOW THE TOP AND BOTTOM GROUPS 
RESPONDED TO ITEMS IN A CAREER EDUCATION INSTRUMENT 





Top 2S 


Bottom 25 




No. of right 


No. of wrong 


No. not 


No. of rteht 


No. of wrong 


No. not 




rtsponm 


rttpoHMS 


mpoffHling 


rtsponsM 


rttponsM 


rttpondinQ 


1 


20 


5 


0 


4 


19 


2 


2 


25 


0 


0 


3 


21 


1 


3 


15 


6 


4 


7 


16 


2 


4 


23 


0 


2 


5 


18 


2 


5 


22 


3 


0 


6 


16 


3 


6 


5 


20 


0 


20 


4 


1 


7 


21 


4 


0 


4 


20 


1 


8 


22 


3 


0 


11 


12 


2 


9 


3 


20 


2 


21 


3 


1 


10 


24 


1 


0 


5 


19 


1 
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In this example, • total of 100 individuals responded to the instrument, so the top 25 and the 
bottom 25 are included in the ar^lysis. For each item, the number of correct responses, the number 
of incorrect responses, and the number not responding are reported for each group (top 25 percent 
and bottom 25 percent). In this example, items 6 and 9 should be changed. Those individuals who 
scored high on this exam tended to answer the items incorrectly, whereas those who scored low 
tended to ansv^ them correctly. Once again, discussing the items with the class will probably 
clarify their responses and present alternative items, 

4, Lengthening the test. Generally, the ler^gth of an instrument is closely related to its reliabil- 
ity. So in developing an instrument you should be aware of the two dimensions of length: number 
of items and necessary completion time. You should probably write twice as many items as you 
plan to use, so that after you have tested the instrument you can substitute the better items for un- 
satisfactory ones. The time required to complete the instrument should not exceed the amount of 
time for which the individuals would nornnally be expected to concentrate and remain still. For 
example, an instrument for secondary students should have a completion time of no longer than a 
regular class period (approximately forty-five to fifty minutes). An instrument for elementary stu- 
dents would normally be shorter. Also, you should allow enough time for most students to com- 
plete the test; naturally some will finish before others, 

5, Providing clear directions. If they are to provide reliable responses, individuals taking a 
career education test must know what is expected of them, what the instrument is designed to do, 
how long it will take to complete, whether they should guess when resporniing to a question, and 
what is to be done with the results. Therefore, the directions must be thorough and detailed. 

Compare the two sets of directions in Figure 17. Which of the two directions would you 
rather receive if you were taking the test? The second set is thorough and explicit and should in- 
form the students what is expected of them. For the sake of clarity, it may even be advisable to 
read the directions aloud before the testing period begirds. 

One consideration in drafting directions is the fact that some school districts and state depart- 
ments of education are requiring explicit procedures for ensuring privacy and the protection of test- 
ing subjects. You should become familiar with these regulatioru before you develop your career 
education instrument. The checklist provided in Figure 18 is designed to guide your development 
of effective directions. 

6, Eliminating ambiguous itams. Ambiguous items are often misurxlerstood, therefore, they 
can cause an instrument to be unreliable. If an instrument is to be reliable, it is essential that the 
items convey precisely what you mean to communicate. Several factors can create ambiguity in 
test items. Four are discussed below: 

• Use of homonyms — A number of words resemble nne another in sound and spelling but 
differ in meaning (such as "sublimate," to divert the energy of an impulse to a higher 
purpose, and "sublimate," to convert a solid into a vapor which on cooling cornienses to a 
solid again). Other words sournJ alike but are spelled differently and mean different things 
(such as "see" and "sea," or "bare" and "bear"). All such words can mislead students 
taking a test and thus should be used with ioi e. 

• Dialect differences — The word "bright," for example, can be used to mean "intelligent." 
but blacks also interpret this word as referring to someone of light skin pigmentation. 
Similarly blacks can interpret the word "bad" as "bad" or as "good" depending upon its 
usage. Additionally, some words do not translate into other languages, for example the term 
"coping" does not translate into a Spanish equivalent. 
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TWO SETS OF DIRECTIONS FOR 
A CAREER EDUCATION INSTRUMENT 



Exampto A 



DIRECTIONS: Plam read Mch item carefully and make your response as accurately 
as possible. 



Example B 



DIRECTIONS: This test is designed to determine your knowledge of the worM of work. 
The results of this test will be used to determine the effectiveness of the social studies in* 
struct ion in which you participated throughout the school year. The results will also repre- 
sent 25 psrcent of your final grade in social studies. Pleese print your name in the space 
at the upper right hand corner of the answer sheet. This mt contains thirty multiple- 
choice questions. Read each question carefully and choose the one best answer to eech 
question. On your smw^r sfwet c!'^rken the space underneath the letter that matches the 
answer you choose. Read the sample question below to see how the answer is to be 
marked on your answer sheet. 

SAMPLE: Which of the following tools would a carpenter be most likely to use? 

a. square 

b. trowel 

c. wire cutters 

d. volt meter 

(a) is the correct answer. On your answer sheet you should darken the space under 
the letter a: 

a b c d 

i II II II 

You will receive two points for each correct answer. Do not hesitate to guess if you 
do not know the correct answer. You will have thirty minutes to complete this test. 
When you have finished, return both the test and ansv«r sheet to your teacher. 
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FiBiirtIS 

A CHECKLIST FOR IMFROVINQ DIRECTIONS 
FOR CAREER EDUCATION INSTRUMENTS 

Ym No 

□ □ 1. Art irKJivkjuait told how to ktomify th«mw!vti on tht 

□ □ 2. Do th« dirtctiont includt wnpit ittim thst illustratt how to mark a mponie? 

□ □ 3. Havt irKlivMuats similar to thoM taking tha tilts baan iKv^ 

raact to tha diractkms? 

□ □ 4. Do tha diracttons axplain tha purpoaa of tha tatt? 

□ □ 5. Do tha diractions ipacify a tima limit for oomplating tha tatt? 

□ □ 6. Do tht diractions stata whara tha indivMual is to rasporxl (a.g., on tha tast or 

onananswar shaat)? 

□ □ 7. Do tha diractions indicata whathar tha indivktual thouM or shoukl not guass? 

□ □ 8. Do tha diractk>ns axplain how tha ratults will ba utilizad and raporttd? 

□ □ 9. If spacializtd (optical scan) rasponst shaats ara raquirad, ara indivkluals told to 

avoid stray marks? 

□ □ 10. Ara individuals told how to changt a rasponst if thay wish to do so? 

□ □ 11. Ara indivkluals tokl how tha tatt will bascortd? 

□ □ 12. Are the santences short and clatr? 

□ □ 13. Arethadirtcttonsraadily visible to tha indivMual taking tha tast? 

□ □ 14. Art tht individuals told what to do whan thty have compltttd tht ttst? 

□ □ 15. Havt you chtcktd with local agtncits rtgarding tha policy for data collection 

from paopit? 

□ □ 16. Havt you chtcktd with statt agtrK:its rtgarding policies for data collection 

from paopit? 
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♦ Compound pr9dic&im or "double bmreled'" $$nt9ncm - An ittm could elicit two reiponm 
when the imtrument allows only one responie per Item, as in the following example: 



I like to keep busy and complete tasks on time. 



The respondent may indeed like to complete a task on time but not keep busy. How is he 
or she to respond? This item should be divkled into two separate items as follows: 



1. I like to keep busy. 

2. I like to complete tasks on time. 



• Inappropriate use of words — Items sometimes contain poor sentence structure or incorrect 
grammar and, at other times, items are stated in a wordy manner. To avoid these pitfalls, 
once you have developed an item, read it aloud, listen or have someone else listen to the 
item. You might also want to have someone read your items to see if you have overlooked 
any errors. It is not unusual for people to overlook their own errors because they know what 
they meant to say. However, someone who has to rely on the printed word when trying to 
understand what you mean is more likely to spot possible problems. 

The six suggestions presented in this unit should assist you in improving the reliability of your 
career education instrument. Once you have refined your instrunrtent by using these techniques, 
you may want to have the reliability checked by the more sophisticated procedures employed by 
an expert. If so, you should be able to obtain assistance from your district or state department of 
education; from colleges and universities, with educational laboratories and centers; or from private 
educational consultants and agencies. While we have noted ways to improve reliability, we should 
not leave this topic without a note of caution. Changes made to improve the reliability of your 
career education instrument affect the validity of the instrument if the item no longer represents 
the area that is to be assessed. (Validity will be discussed in the next chapter.) Therefore, changes 
made to improve reliability should be made with care. 
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DETERMINING VALIDITY 



The adage that states "you cannot judge a book by its cover" could well be extended to reed 
"you cannot judge a test by its title/' For example, an instrument entitled ''Career Decision Mak- 
ing" n^y not actually constitute a test on this activity. It may be simply a collection of quickly 
assembled items loosely related to the topic and rushed into use. This section of the handbook is 
designed to help you understand what is meent by the term validity, how you can check your instru- 
ment's validity, and how you can improve the valkJity and therefore the quality of your own career 
education instruments. 



What Is Meent by the Term Validity? 

Validity refers to the accuracy with which a test meesures what it is supposed to meesure. The 
literature on testing generally describes four major types of validity: Mntent, concurrent, predictive, 
and construct. You need to determine which of these types are most appropriate for testing your 
instrument. 

Content vBlidity refers to the representativeness of the test items and their relationship to the 
topic being measured. For example, a test on decision making would not be able to include a// the 
information needed by a student, but it shoukJ contain an adequate sampling of information from 
the content area. Furthermore, the test shoukl exclude information on tangential areas, such as 
the world of work, economic urnierstanding, or self-awareness. 

Concurrent validity determines how much agreement exists between a score on a particular 
instrument and a score obtained from other acceptable instruments, bodies of knowledge, or skills 
in the same content area. An example of concurrent valMity woukJ be the similarity between the 
students' scores on an instrument on work habits arKj their actual habits observed on the job. 

Predictive validity is the ability of an instrunient to predict future performance based on pre- 
determined criteria. For example, scores on the Strong-Campbell Vocational Interest Test have 
been shown to predict later career choices. The major difference between predictive and cofKur- 
rent validity is the time element: concurrent validity is demonstrated in the present, while predictive 
validity is demonstrated in the future. 

Construct validity "proves" certain hypothetical relatk>ns. The construct validity of a career 
interest instrument, for example, determines what extrinsic factors (such as general knowledge of 
careers, attitudes toward work, and views toward the locus of authority for personal development) 
account for the variations in the test scores. The hypothetical nature of construct validity differen- 
tiates it from other types of validity and places it among the concerns of researchers and professional 
test developers; therefore, construct validity will not be discussed in this handbook. 

This section has discussed three different ways to determine the validity of your career educa- 
tion instrument. The next section presents specific suggestions for improving the validity of your 
instrument. 
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How Can I Check a Career Education 
Initrunrwnt for Validity? 



This section will discuss how you can determine three of the preceding types of validity: 
content, concurrent, and predictive. 

Content validity can be evaluated by having experts review and rate the test items for relevance. 
A panel review by three to nine authorities is desirable. For example, if you were working on an 
instrument gauging economic understanding, economists should review the test to determine if it 
does in fact measure economic understanding. In this review process, the experts should be asked 
to respond to such questions as: 

• Are all the items related to economic understanding? 

• Do the items provide an adequate assessment of econom ic awareness? 

• What items should be added to make the instrument more representative? 

• Does the title of the instrument accurately represent the material included? 

A sample form that could be used in this process is presented in Figure 19. 

Concurrent validity can be determined by comparing the results obtained from your instrument 
with an independent assessment of a different, previously validated instrument. These two measures 
should be obtained at the same point in time. If there is agreement between these two separate in- 
struments, your instrument is said to have concurrent validity with the previously validated one. 

An easy method for establishing concurrent validity is to arrange the students' names in sequen- 
tial order from the highest score to the lowest on each of the two separate instruments. Once the 
students have been ranked in order, you should proceed to compare the results of your instrument 
with the results of the previously validated instrument. In making this comparison you would use 
the same procedure suggested for determining the equivalence of two forms of the same test that 
was discussed in the reliability section (see p. 24). In determining concurrent validity the names of 
each student on the previously validated test would be placed in rank order (from highest to lowest) 
and the appropriate rank for each student would be placed in a column to the right of the appro- 
priate name column. The rank order for each student on your test should then be placed in a 
second column to the right of the appropriate name (see Figure 14, p. 25). You can then proceed 
to compute the concurrent validity following the same formula and procedures discussed for the 
equivalence reliability. The closer this number is to 1.00 the more valid your test is said to be. 

Predictive validity is determined by comparing the present scores on an instrument with future 
scores from a different instrument. For example, let us assume that you have developed an instru- 
ment on occupational awareness and have administered it to twelfth grade students. In general, 
those who are more knowledgeable about occupations tend to make better career decisions, so you 
want to determine if the students who scored higher on your instrument will be satisfied with their 
educational and career choices. Therefore, an independent test of job satisfaction administered to 
these same students at a later time could be used to determine if the occupational awareness instru- 
ment did, in fact, forecast job satisfaction. 

In general, you can determine predictive validity in much the same way that you determine 
both concurrent validity and equivalence reliability. First, you order the names, by score, from 
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A SUGGESTED FORM FOR A PANEL REVIEW OF 
AN ECONOMIC UNDERSTANDING INSTRUMENT 



Dl RECTIONS: From your experience, please judge whether the items included on the 
Economic Understanding Instrument are necessary for high school students to know in 
order to make an effective transitk>n from school to work. In making this judgment, pl^ 
check your opinion (yes, no, or undeckled) on the scale to the right of each numbered 
item. Any comments regarding this item shoukl be written in the space to the far right. 



Comments 



Itsm 


Ym 


No 


UndMkM 


1 


□ 


□ 


□ 


2 


□ 


□ 


□ 


3 


□ 


□ 


□ 


4 


□ 


□ 


□ 


5 


□ 


□ 


D 


6 


□ 


□ 


□ 


7 


□ 


□ 


□ 


8 


□ 


□ 


□ 


9 


□ 


□ 


□ 


10 


□ 


□ 


□ 



In the following space (and, if necessary, on the back of this page) please comment on the 
entire test. 

1 . Is the content of this instrument representative of the knowledge required of high 
school students to make more effective transitions from school to work? If not, 
what else should be added? 



2, Are all the items related to economic understanding? If not, which ones should be 
eliminated? 



3. Does the title of this instrument represent the material included in the individual 
items? If not, what should the instrument be called? 
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highest to lowest on the first (occupational awareness) instrument and place the nunnber of the 
appropriate rank order in a colunnn to the innnnediate right of each nanne. This colunnn should be 
in correct nunnerical order. You then proceed to rank the students' names in order (from highest 
to lowest) according to their scores on the second (job satisfaction) test. The rank of each student 
on the job satisfaction test is then placed in a second column to the right of the appropriate name 
(see Figure 14, p. 25). Again, the formula and procedure for determining the predictive validity of 
a test is the same as that suggested for equivalence reliability (see p. 24). Remember, if you want to 
determine predictive validity you must plan ahead and retain test scores on one test so that you can 
compare them with scores from another test at some future point in time. 



How Can I Improve the Validity of a 
Career Education Instrument? 

This section provides six practical suggestions for improving the validity of your career educa- 
tion instrument: 

1 . Prior to revising the instrument, review the information on the goals and objectives of 
the career education topic to be assessed. 

2. Use a planning matrix to develop the instrument. 

3 Research occupations so that you can develop items based on the different jobs that 
various individuals perform. 

4. Develop twice as many items as you intend to use m the instrument. 

5. Lengthen the instrument. 

6. Have experts review the instrument for relevance to the topic being assessed. 

1. Reviewing information on the goals and objectives. Please refer to Unit 1, "Planning and 
Designing," pp 9-10- 

2. Using a matrix. Using a planning matrix (see Figure 7, p. 17) to develop a career education 
instrument is likely to increase the validity of the instrument for several reasons. First, the matrix 
facilitates the development of a clear plan for the instrument. Second, the matrix enables you to 
check the relevance of test items to the overall plan and thus keep the test "on target." Third, 

the matrix facilitates the selection of a representative number of items to achieve the purpose of 
the instrument And fourth, the matrix enables you to regulate the kinds of responses called for 
by the test items, this advantage is important because items that require an individual to respond 
in ways other than memorization tend to constitute more valid instruments. Using a planning 
matrix, then, can be an important aid to improving validity in an instrument. 

3. Researching occupations. Educators seldom have sufficient information about occupations 
to develop valid career education instruments. Therefore, it is frequently necessary to research 

the kinds of work required in various occupations. One of the most obvious ways to do this is to 
talk to someone in that occupation. Another way is to read about various occupations. Fortunately, 
several readily accessible reference works can help supply this background information. 

• The Dictionary of Occupational Titles (1965) 
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• The OccupetiormI Outlook Handbook (revited every few years) 

• Directory of Tmklnventorim Volume I (1974), Volume II (1975), and Volume III (1976) 

The more precitely your items present the real world of work, the more valid your instrument is 
likely to be. 

4. Devising sufficient items. A good rule of thumb to follow when developing an instrument 
is to devise twice as many items as you intend to use. If you can develop a "pool" of items on the 
topic you wish to assess, you can then draw upon these items to develop an instrun>ent that is nfK>re 
representative of the topic. The fewer the items that you have, the less likely that instrument will 
assess the topic thoroughly. You can then categorize these test items by using a planning matrix 
(such as that illustrated in Figure 7 on page 17) in order to determine if appropriate emphasis has 
been placed on each of the learning tasks and domains included in the instrument. 

5. Lengthening the instrument. Though it is overlooked, lengthening the instrument is a good 
way to improve its validity. Including more items allows you to sample more of the learning tasks 
that need to be included. Conskier the example of an instrun>ent on work habits that consists of 
five learning tasks (the state of Florkia, 1973): time utilization, responsibility, safety and neatness, 
use of instruction, and attitude toward the job. In an instrunient consisting of twenty*ftve items 
with equal emphasis on each learning area, no more than five items could be included for each 
learning task. With this limited number of items, the results may not reflect an individual's work 
habits. Doubling the number of items for each learning task increases the likelihood that the results 
would more accurately reflect the individual's vork habits. Lengthening the instrument, then, can 
increase its valklity. However, also keep in mind the ler>gth of time it takes to administer a test- 
completion time often limits the number of items that can be included on an instrument. 

6. Reviewing by experts. Once you have developed the initial draft of a career education 
instrument, you should have knowledgeable people (e.g., teachers, subject matter specialists, business 
persons, etc.) review the proposed test. This review, sometimes referred to as assessing the "face" 
validity, could follow the same format as that suggested for content validity (see Figure 19, page 35). 
A second review technique is having the reviewers classify the test items accordirtg to the learning 
tasks or classes of the appropriate educational domain. Educators may classify more effectively the 
items according to the class of the educational domain being considered, whereas business persons 
and other noneducators may classify more effectively the items according to the learning tasks. In 
classifying test items you could use a modification of the planning matrix (page 17) and ask each 
person to write the number of the item under the appropriate heading. An example of how this 
planning matrix could be modified to record the number of test items by class of educational domain 
is presented in Figure 20. An example for classifying items by content area is shown in Figure 21 . 
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A SAMN»LE FORM FOR RECORDING TEST ITEMS BY CLASSES 
OF THE COGNITIVE EDUCATIONAL DOMAIN 
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A SAMPLE FORM FOR HECORDINQ TEST ITEMS BY CONTENT AREA 
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ELIMINATING STEREOTYPES 



Career educeton in particuler should be corKemed with breeking down i t ereotype t eo thet all 
individuals will be encouraged to pursue meaningful careers. As with any curriculum, cereer educe- 
tion materials-not only instruments but also accompanying manuels, resources, and reporting 
formats-should be free from stereotyping of any type. Most publishers and agencies in the field of 
education have guidelines for authors to follow in prepering materiels tfiet ere free from stereotyp- 
ing. This section of the handboolc, based upon these widely accepted guidelines, is designed to help 
you reduce sex and ethnic stereotyping in cereer education instruments* 



Are My Career Education Instruments 
Free of Stereotypes? 

This section presents five practical suggestions for reducing stereotypes in career education in* 
struments. 

1 . Watch your pronouns. The pronoun "he" has often been used in e generic manner; todey, 
however, this usage is considered abrasive and tends toward the continuation of sexual stereotypes. 
One way of eliminating the stereotype is to substitute the terms "he/she/* "her^im/' "himself or 
herself/' "hers or his/' etc. In addition, you should vary the sex of the pronoun ghren first (i.e., 
female or male). This remedy should not be overused, however, as it can become obtrush^, A less 
obtrusive way of eliminating the stereotype is to change singular nouns to plural nouns and then 
use the accompanying plural pronouns "they/' "theirs/' etc. There ere also other ways to avoid the 
generic "he": the noun can be repeated, and sentence structure cen be changed to evoid the dilemma. 

2. Use neutral terms in describing various occupctiora or careers. Many words referring to 
workers and professionals reflect a male<lominated society. These words include "businessman," 
"chairman," "mailman," "fireman." Listed below are neutral alttmatives for several job deslgnattons: 



Previous Usage 


Neutral Term 


Businessman 


Businessperson 


Chairman 


Chairperson 


City hostess 


Goodwill ambassador 


Fireman 


Fire fighter 


Makl 


Houseworker 


Mailman 


Mail carrier 


Manpower 


Labor force 


Policeman 


Police officer 


Salad girl 


Salad maker 


Stewardess 


Flight etterxlent 



The U.S. Department of Labor has published a document entitled Job Title Revisions to Elimi- 
nate Sex- and Age- Referent Language from the Dictionary of Occupational Titles (1975). If you are 
unable to think of a neutral term to describe an occupatk>n, this publication may be helpful. 
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3. Include in ittrm on tht instrumtnt nanm that are difftrtnt from Anglo in origin. The 
tradition of using only Anglo names in educational materials has been cited as one reason that 
minorities may not have seen the full range of career opportunities that others have seen. There- 
fore, it is important that career education instruments use names that are of obvious ethnic origin, 
such as: 



Coretta Juan 

Yumiko Nariyo 

Mohammed Rao 

Inez Gilda 

Zuynk Canta 

Youngja Sugil 

Shoshana Samuel 

Undoubtedly, you will be able to list many others simply by drawing upon the names of celebrities, 
colleagues, and friends. 



4. Include items that reflect a variety of life stylet, personalty typea, socio^«M>nomic levtis, 
cultures, and handicaps. Educational materials have long mirrored the "middle class" value system 
of Americans. These materials invariably depicted a nuclear family with the father as breadwinner 
and the mother as homemaker. Seldom were extended families or "broken homes" depicted. 
Similarly, little attention was given to different cultures or socioeconomic levels. Furthermore, 
handicapped persons never appeared in educational materials. Many individuals using these mate- 
rials naturally found it difficult to identify with the characters and consequently their educational 
performance was hampered. Therefore, career education instruments, whenever possible, should 
illustrate a variety of life styles, personality types, socioeconomic levels, cultures, and handicapping 
conditions. In presenting more varied life styles, however, you must be careful to avoid perpetuat- 
ing sex-linked or ethnic-linked stereotypes (e.g., outgoing males, shy females, black broken home). 

5. Include some items that depict males in traditionally female rol« and occupations and 
females in traditionally male roles and occupations. The news is full of accounts of both men and 
women excelling in activities that were once thought to be the exclusive domain of the "opposite" 
sex. Consequently, illustrations of men and women in wider ranges of roles and occupations not 
only encourage equal rights and opportunities, but also reflect today's world. Career education 
materials, because of their special impact on future conditions within professions, should depict a 
broad range of opportunities for both men and women. The materials should also include blacks 
in executive or supervisory roles and role "reversals" (e.g., white working for black, man working 
for woman). 
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DEVISING THE FORMAT 



First impiwions are lasting. Gttting btyond a nagntivt first imprmion to a more thorough 
and fair appraisal ganarally takn quite some time. If the first impressions generated by a career 
educetion instrument are negetive, the persons taking the test mey not begin to perform at their 
highest levels. Therefore, in order to devek>p cereer educatton instruments that provide accurate 
and reliable results, it is advisable to devise a format that makes a good first impres»on by means 
of a clear, neet, k>gk:ai, and efficient layout. 



How Can I Improve the Formet of My Career 
Education Instrument? 

This section is designed to provMe a list of four suggestk>ns for designing or improving the 
format of your career educatk)n instrun^t. 

1. Limit the number of differem types of Items on your cereer education instrumem. Each 
time that you change the type of item included on your cereer educetk>n instrument (e.g., true*false, 
multiple choice, short answer, matching, essay, rating scale, etc.), you cen disorient the indhriduals 
taking the test. Therefore, it is desirable to limit the number of different types of items on tests 
and make deer transition from one to another when necessery. New directk>m shoukl be included 
whenever the format changes. A good rule of thumb is to use only two or three different types per 
instrun^ent. Similarly, it is helpful to ^up all items of the same type together. For example, on 

an instrument with three types of items, keep all the multiple choice items together, all the matching 
items together, and all the true-false items together. This grouping will eneble the examinees to lose 
less time reeding directions and reorienting themselves to the format, and thus alk>w more time for 
responding to the items. 

2. Use large type and dear illustratiom when prepering the i ns trument. When taking a career 
education test, indivkJuals should have to cortcern themselves only with reeding the question and 
making an appropriate response. It is counterproductive and in fact, unfair to require these indi- 
viduals to puzzle over blurred words or complicated illustretions in addition to providing a response 
to the item. Therefore, you should always use large, cleen type or print when preparing a career 
education instrument. This suggestion is particularly important as you will subsequently have to 
reproduce the instrument. The higher the quality of the master copy, the higher will be the quality 
of subsequent reproductions. Make responses as short as possible, however, in doing so do not 
sacrifice clarity. 

Illustrations also need to be clear. Unnecessary information and detail shoukl be eliminated 
so that individuals can quickly focus their attention and respond to the item. In addition, an illus- 
tration should appear on the same page as the items that refer to it so that individuals need not lose 
time flipping pages back and forth. 

3. Keep all possible responses to an item on the same page. An important guideline to remem- 
ber when preparing a career education instrument is: simplify the response format. One way to do 
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this is to include all possible responses to an item on the same page as the item. Imagine the dif- 
ficulty the respondents would have if they had to flip back and forth between pages in order to 
consider multiple choice items C and D or to complete matching items in an exercise. 



4. Prepare an attractive page layout. The logical sequence of exercises and the attractiveness 
of entire pages are focal points for preparing an attractive layout. If blanks are required for com- 
pleting items, they should be of uniform length. If the blanks are placed to the left of the items, 
they will not only add regularity and neatness to a prominent visual portion of the page but will also 
make scoring easier. Similarly, placing all possible responses directly beneath an item will improve 
the attractiveness of the layout. It is necessary, though, to consider the responses for all of the items 
on a single page (both multiple choice and matching, for example) and to select the pattern that 
allows for consistency as well as neatness of design. 

The appearance of the entire page is very important. Items should be spaced so that the page 
appears uncluttered. Ideally, margins should be uniform on both sides of the page. The balance 
between printed material and unprinted space should be even and symmetrical. Even after following 
these guidelines, however, you should have a group of teachers review the layout and offer their 
suggestions for improvement. 
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GAUGING READABILITY 



An often cv«ricok«d crittrion for judging your carttr cducstion instninngnt U whtthtr mpond* 
entt art abta to undtmand wtM you hmm writttn. This critgrion it pgrticulariy importam whan 
tha raspondants ara of vaatly diffaram backgroundi and abilitiaa. Tha inatrumam naad not ba 
diractad at tha towatt la^alt of achtavamant in ordar to taai all tha ra^iondants. 



How Can I Imprava tha RaadMHty of a 
Caraar Eduoatkwi Inrtni i nati t ? 

This saction will praaant th ^ suqgaations for improving thg rgadabilhy of a caraar aducation 
instrumant: (1) davalop tha insuur ^ so that it prograMt from a faw raiativaiy simpla itanrv at 
tha beginning to mora complax itarm .^tar on, (2) conduct readability taats, and (3) pilot tast tha 
instrumant with individuals similar to thoaa fbr whom you ara preparing tha instrumant. 

1. Begin with retotively eaiy items* Accepted principles of eduartionel psychology dictate 
that teechars should program from tha familiar to tha unfamiliar and from tha easy to the difficult. 
These principles also epply to the davaiopmant of oreer education instrunrwvts. Therefore, you 
should gauge the level of difficulty for all of the items, either by claasifying them yourself (e.g., es 
easy, moderate, or hard) or by conducting a pilot tast (as deecribed in s u gges ti o n 3). After gauging 
the difficulty of the items, you could then orgenize them by beiH'^i'HI with eesiar items and pro- 
grassing to the more difficult ones. 

2. Conduct raadabiltty tests. Tha reading level of e career education instrument cen be ob* 
tained by using the reedability formulas found in numerous education texts. Dale and Seels (1971) 
have de 5loped a comprehensive bibliography on raedebility measures. Other sources include Flesch 
(1951), Bormouth (1969), and Chall and Dale (1948). Howavar, it should ba noted that formulas 
alone do not furnish exact answers. You should still conduct a pilot test of an instrument prior to 
its use. 

3. Pilot test the instrument. The reeding level of a career education instrument should not be 
condescending o*^ childish; nor so difficult that it discourages examinees and thus undermines the 
results obtained from the instrument. One way to check the readability is to conduct a pilot test 
of the instrument by assembling a reprasentativa group of individuals and asking them to identify 
the major problems they find in understanding the instrument. This method is lass sophisticated 
than the complex methods noted in suggestion 2. This group should represent es wkle a renge of 
academic, sock>eoonomic, and cultural levels as possible and shouW include both males and females. 
After each of the reviewer s has marked all the difficult words and items, ask the group to discuss 
the instrument on an item-by-item basis arNl indicate any problems that they find. In working with 
younger groups, you sho>^kJ lead tha students through the process orte sectk>n at a time. If you ask 
the individuals to indicate their reactions indeperKiantly of one enother, you minimize peer group 
pressures; thus you imp>rove your chances of noting different points of view and of provkiing a more 
realistic review of the instrument. 
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EXAr^ilNiNG YOUR CAREER EDUCATION INSTRUMENT 



Th« prtctding units of thit hindbook h«v« presented six topics releted to improving career 
education instruments: planning and designing, checking reliability, determining validity, eliminat- 
ing stereotypes, devising the format, and gauging reedebility. These units have provided you with a 
basic awareness of some problems encountered in developing instnjments as well as some techniques 
for minimizing these problems. With this basic awareness, you are now reedy to examine the instru- 
ment that you have developed. Your goal is to establish confidence in your instrument, two arees 
of major concern in such a task are those of validity and reliability. As a start, let's take a look at 
the content validity and internal consistency reliability of your instrument. 



Determining Content Validity 

First, let us detenrnine the instrument's content validity. In making this determination it will 
be necessary for you to develop and utilize the format and procedure for conducting a panel review. 
A sample format was suggested on page 35. Review this format and procedure and then modify the 
format to meet the needs for your instrument. You will probably have to change a few words in 
the directions and questioni asked and adjust the form for the number of items on your instrument. 
Once you have made these nrMxJifications, select a panel of individuals that are considered knowledge- 
able of the area to be assessed. It would be desirable for you to include persons who are knowledge- 
able from education as well as personnel from business, industry, and labor. 

In conducting the panel review, it is generally desirable to have each member of the panel, 
review the instrument independently. Once they have completed their reviews and complete! the 
panel review forms, ask panel members to share their reactions with each other. This discussion 
should provide the information you need to determine how well your instrument measures what 
you had intended for it to measure. If most of the panel members are basically pleased with the 
instrument you are ready to proceed. However, if the panel is divided in its opinion or most of 
the panel members are unhappy with the instrument, you will need to do some addition* I work on 
the instrument before moving on to the next step. If additional work is required you should con- 
duct a second panel review of the revised instrument to make sure necessary improvements have 
been made before proceeding to next step: determining internal consistency reliability. 



Determining Internal Consistency Reliability 

If the panel has decided that your career education instrument measures what you intended 
for it to wf dsure, you are ready to determine its internal consistency reliability. In making this de- 
termination it would be advisable to administer your instrument to a minimum of thirty students. 
(Ideally, you should check the reliability of an instrument with ten times as many students as you 
have items on a test. For example, if your instrument has twenty-five items, it would be desirable 
to administer it to 250 students.) You also need to make sure that these students are representative 
of the intended population for the instrument. Every effort should be made to see that these stu- 
dents represent all cultural, economic, educational, and ethnic backgrounds. The sample should also 
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include the disadvantaged, the handicapped, and equal number of representatives of both sexes. 
After a sufficient number and variety of students have been selected and the test has been adminis- 
tered you can proceed to determine its reliability. 

First, it will be necessary to divide the test into two parts. An acceptable way of doing this is 
on the basis of odd- and even-numbered items. Score each of these two sections, determining a 
final score for each. Once these two scores are computed for each student you are ready to begin 
computing the internal consistency reliability. 

Second, arrange the tests in rank order from the highest score to the lowest score for each 
student on the even-numbered items. Now, prepare a list of these students' names in rank order 
from the highest to lowest score and number them accordingly. 

Fourth, merge the information on these two lists into one form as shown in Figure 12 in the 
reliability section. A sample form for your instrument is shown in Figure 22. 

Fifth, compute the internal consistency reliability using the formula: 

T= S 

N (N-1) 

Substitute the appropriate figures for your instrument and complete the compjtation. A sample 
problem of this type is shown on pages 22-23. The value you compute for T should be between 
0.00 and +1 00. If this is not the case, you have made an error and you should recheck your com- 
putations. If your answer is within the range mentioned above, check with t juidelines presented 
on page 19 for interpreting reliability coefficients. How acceptable is your reliability coefficient? 
If It is not at least .80 (see Figure 8) you should give additional thought to revising your instrument. 
Specific suggestions for increasing the reliability of tests are presented on pages 26-32. 



Figure 22 

A SAMPLE FORM FOR RECORDING SCORES 
WHEN DETERMINING INTERNAL CONSISTENCY RELIABILITY 



fg^^g Rank on Odd- Rank on Even- 
Numbered Items Numbered Items 

1 

2 

3 

4 

5 

6 

7 

8 

9 
10 
11 
12 
13 
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SOME CONCLUDING R&MARKS 



Developing sound evaluation instruments is challenging. Educational measurement is a relatively 
new diacipline, and although it is gaiiiing sophistication and acceptance, it will always be impossible 
to gather all of the information svailable in any attempt to measure human activity. The task of 
developing an instrument for the field of career education is further complicated by the fact that 
career education is also a new discipline, and it is still developing the precise definitions and concepts 
that characterize the more established scientific and technical fietdt. 

A number of books and articles have been prepared to assist in the measurement of human 
activity. Certainly these materials, although not developed specifically for career education, have 
relevance to career education and a number of suggestions can be obtained through their use. We 
have cited a number of sources throughout this handbook, and while it wouW be difficult to list 
all of the materials that would be useful to you, we do think two books of general use nrwit your 
attention: 

T9acher'M9de Tests (Green, 1975) 

Measuring Educational Achievement (Ebel. 1965) 



Finally, this handbook was designed to help allay some of your fears and concems about develop- 
ing evaluation measures. Because of the diverse needs of each career education program, the 
handbook was designed to identify major basic issues and offer suggestions for you to consider when 
developing measures. Specifically, it addressed the topics of improving the planning and designing, 
checking reliability, determining validity, devising the format, examining the readability and eliminat- 
ing stereotyp'TS of career education instruments. Another handbook in this career education measure- 
ment series. Career Education Measures: A Compendium of Evaluation instruments, also provides 
examples of other career education instruments that have been developed by other individuals. Thus, 
the materials contained in this handbook and the suggested references together shouW provkle you 
with a foundation for designing improved career education measures. By using these sources and 
others that you may be familiar with, you should now be ready to develop and improve career educa- 
tion measures to meet your needs. We think you will find this to be a challenging and rewarding 
activity. 
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